Goto

Collaborating Authors

 Maury County


Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles

Truong, Kimberly Le, Fogliato, Riccardo, Heidari, Hoda, Wu, Zhiwei Steven

arXiv.org Artificial Intelligence

Current benchmarks for evaluating Large Language Models (LLMs) often do not exhibit enough writing style diversity, with many adhering primarily to standardized conventions. Such benchmarks do not fully capture the rich variety of communication patterns exhibited by humans. Thus, it is possible that LLMs, which are optimized on these benchmarks, may demonstrate brittle performance when faced with "non-standard" input. In this work, we test this hypothesis by rewriting evaluation prompts using persona-based LLM prompting, a low-cost method to emulate diverse writing styles. Our results show that, even with identical semantic content, variations in writing style and prompt formatting significantly impact the estimated performance of the LLM under evaluation. Notably, we identify distinct writing styles that consistently trigger either low or high performance across a range of models and tasks, irrespective of model family, size, and recency. Our work offers a scalable approach to augment existing benchmarks, improving the external validity of the assessments they provide for measuring LLM performance across linguistic variations.


GM to double revenues and increase capacity for EV assembly at factories

USATODAY - Tech Top Stories

General Motors secured its pivot to a high-tech future Wednesday by announcing plans to convert more assembly plants in North America to make electric vehicles by the end of the decade and to double company revenues by that time as it unveils more software and new EVs. GM leaders are expected to reveal new technology, such as an advancement to the automaker's hands-free driving system, as well as more EV products, including a new Chevrolet SUV EV priced at $30,000, during its annual Investor Day presentation, starting at 1 p.m. ET. "Our early investments in these growth trends have transformed GM from automaker to platform innovator, with customers at the center," said CEO Mary Barra. "GM will use its hardware and software platforms to innovate and improve their daily experience, leading everybody on the journey to an all-electric future." Barra emphasized that no hourly workers will lose their jobs in the transition to electric and GM continues to hire salaried employees who have a background in technology and digital software. "Late last year we hired 3,000 employees and this year alone we've hired 8,000 salaried workers especially in the technology, digital and the software space," Barra said.


Janice: Excited for eclipse

FOX News

I was 8-years-old and remember being both terrified and intrigued about something that was being talked about everywhere. This wasn't a storyline out of a science fiction movie or novel, this was real, and happening here on Earth. Millions of people were going to witness something that maybe happens a couple of times in our lifetime: A total solar eclipse. Our teachers were planning lessons about this incredible celestial event. Chalkboard diagrams, planetary mobiles and handmade viewing devices were being created out of shoe boxes.